Open
Conversation
…n last CSV column ### What changes were proposed in this pull request? This PR fixes an issue where the CSV reader inconsistently parses empty quoted strings (`""`) when the `escape` option is set to an empty string (`""`). Previously, mid-line empty quoted strings correctly resolved to null/empty, but the last column resolved to a literal `"` character due to univocity parser behavior. ### Why are the changes needed? To ensure consistent parsing of CSV data regardless of column position. ### Does this PR introduce _any_ user-facing change? Yes, it fixes a bug where users were receiving incorrect data (a literal quote instead of an empty/null value) for the last column in a row under specific CSV configurations. ### How was this patch tested? Added a new regression test in `CSVSuite` that verifies consistent parsing of both mid-line and end-of-line empty quoted fields.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR fixes an issue where the CSV reader inconsistently parses empty quoted strings (
"") when theescapeoption is set to an empty string ("").Previously, if
escape=""was used, mid-line empty quoted strings were correctly resolved to an empty string, but the last column resolved to a literal"character. This occurred because Spark Mapsescape=""to\u0000, which univocity’s parser relies on. At the end of a line, without a trailing delimiter, univocity misinterprets the second"as an escaped quote rather than a closing quote.The fix introduces a post-processing step in
UnivocityParser.parseLineto detect this specific condition (a single quote character as the last token whenescapeis\u0000) and replace it with the configuredemptyValueInRead.Why are the changes needed?
To ensure consistent parsing of CSV data regardless of whether an empty quoted string appears in the middle of a line or at the end of a line.
Does this PR introduce any user-facing change?
Yes, it fixes a bug where users were receiving incorrect data (a literal quote instead of an empty/null value) for the last column in a row under specific CSV configurations (e.g.
escape="",quote="\"",sep=";").How was this patch tested?
Added a new regression test in CSVSuite.scala:
"SPARK-46959: CSV reader reads data inconsistently depending on column position"The test verifies that an empty quoted string behaves identically in the mid-line position (column c) and the end-of-line position (column d) when configured with
escape=""andnullValue="".Verified that both
CSVv1SuiteandCSVv2Suitepass without regressions.Was this patch authored or co-authored using generative AI tooling?
No